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Abstract. Let B n = S n (S n + QnTjv) -1 , where S„ and Tjv are two independent sam- 
ple covariance matrices with dimension p and sample sizes n and N respectively. This 
is the so-called Beta matrix. In this paper, we focus on the limiting empirical spectral 
distribution function and the central limit theorem of linear spectral statistics (LSS) of 
B„. Especially, we do not require S n or Tjv to be invertible. Namely, we can deal 
with the case where p > max{n, JV} and p < n + N. Therefore, our results cover many 
important applications which cannot be simply deduced from the corresponding results 
for multivariate F matrices. 



1. Introduction. 

In the past two decades, more and more large dimensional data sets appear in scientific 
research. When the dimension of data or number of parameters becomes large, the classi- 
cal methods could reduce statistical efficiency significantly. In order to analyze those large 
data sets, many new statistical techniques, such as large dimensional multivariate statisti- 
cal analysis (MSA) based on the random matrix theory (RMT), have been developed. In 
this paper we will investigate a widely used type of random matrices in MSA which are 
called Beta matrices. 

Firstly we introduce some definitions and terminology associated with Beta matrices. 
Let X n = (xij) pxn , where {xij} are independent and identically distributed (i.i.d.) ran- 
dom variables with mean zero and variance one, and S n = n~ 1 X n X* which is known as 
the sample covariance matrix. Here the superscript * stands for the complex conjugate 
transpose of a matrix. Similarly let T^r = A r ~ 1 XjvX^ r be the other sample covariance 
matrix, where Xjy = (^-ij)pxN and {^ij} are i.i.d. mean zero and variance one random 
variables. The Beta matrix is defined as 



I.I) B n = S n (S n + a n T 



Nj 
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where a n is a positive constant. For any n x n matrix A with only real eigenvalues, we 
denote F A as the empirical spectral distribution function (ESDF) of A, that is 



1 n 



n . 

where denotes the i-th smallest eigenvalue of A and /(•) is the indicator function. In 
this paper we focus on the limiting ESDF and the central limit theorem (CLT) of LSS of 
B n - 

One motivation to study Beta matrices is that their ESDFs are very useful in MSA, such 
as in the test of equality of k(k > 2) covariance matrices, multivariate analysis of variance, 
the independence test of sets of variables, canonical correlation analysis and so on. There 
is a huge literature regarding this kind of matrices. One may refer to [HJ dj QUI E2] for more 
details. For pedagogical reasons, we provide one statistical application of Beta matrices 
as follows. 

Let {z[ , ...,z„ } be an i.i.d. sample drawn from a p-dimensional distribution and 

(2) (2) 

{z\ ,...,z N } be an i.i.d. sample drawn from the other p-dimensional distribution. 
Suppose fii = Ez^ = and E.; = Yarz(\ i = 1,2. Write zp = E^ 2 X(.j) and 



z~p = E^ 2 X(.j) where X(.j) is the j-th column of X n (Xjv) and E^ 2 is any 

square root of Ej . We wish to test 

H : Ei = E 2 v.s. Hi : E x ^ E 2 . 

This is one of the most elementary problems in MSA, for which there are lots of test 

statistics. If we write z£ } = n" 1 £™=i and Z S? = N ~ X £"=i z f \*f then 

all the following Lj, j = 1, 2, • • • ,5 are the most frequently used test statistics for Hq (see 



Chapter 8 in [12 
(1.2) 



| Z (l)|n I Z (2) ,iV 

Li = log ' " 1 1 7' -= / (nlog(x/c n )-N\og((l-x)/c N ))dF B "(x), 



L 3 = log|z«(Z« +a n Z% ) )- l \ = p J logxdF B "(x), 
L 4 = tr to (z« +a n zg ) )~ 1 J = pj xdF B "(x), 

L 5 = c n tr (z$ ( Cn Z« + CNZ^y 1 - lj + c N tr (z$ (c n Z« + c N Z% ] 

= CnP j (C^X - lfdF B "(x) + CNP J (C N \1 - X) - l) 2 dF B "( X ), 

where c n = n/(n + N), cn = N/(n + N) and a n = N/n.. Apparently all the above test 
statistics are linear functionals of the ESDF of Beta matrices B n , which are all the LSS 
of B„. It is already well-known that the classical limit theorems for those LSS are not 



CONVERGENCE OF ESDF OF BETA MATRICES 



valid when the dimension is large. So it is crucial to investigate the sequence {i^ Bn } in 
the large dimensional case. The following result tells us the limiting behavior of {F Bn } 
as p,n,N — > oo. 

Theorem 1.1. (Limiting spectral distribution function (LSDF)) Assume on a common 
probability space: 

(i). For each i,j,n, Xij = x n ij are i.i.d. with Exn = 0, E|xn| 2 = 1. 



1. 



(ii) . a n — > a > and y n = p/n — > y > 0. 

(iii) . For each k,l,N, stjy = xjv/y are i.i.d. with Esn = 0, E|xn| 2 

(iv) . Y N = p/N -> Y > and ^ -> Jg, e (0, 1). 

(v) . sup n E|xn| 4 < oo and sup^r E|xn| 4 < oo. 

T/ien uwi/i probability 1, F B "^F weakly, where F is a non-random distribution function 
whose density function is 



V((a(l-y)-l+i/) 2 +4 a )(f r -t)(f-f ; ) 
0, 



when ti < t < t r ; 
otherwise, 



where t[,t r = (^ 2a ^ ^^^^yj,^^^^ yY+Y ^j . In addition, when y > 1, F(t) has a 
point mass 1 — 1/y at t = 0; when Y > 1, F(t) has a point mass 1 — 1/Y at t = 1. 

Remark 1.2. Condition yY / [y+Y) < 1 is to guarantee that the random matrix S n +a n T n 
is invertible almost surely otherwise the dimension p could be eventually larger than the 
number of observations n + N . This would imply that S n + a n TN is singular. Condition 
(v) gives us the a.s. bounds of the limit of the smallest and largest eigenvalues of the 
random matrix S n + a n Tjy since by the definition o/B n we can rewrite 



1 



n + N 



X\n 



n-LN 



2K11 



Here 



/ ( n+N 
I n 

V 
V 



Xpn ^Spl 



Xln ^11 



n+N 



(n+N)a n 
N 



V 



(n+N)a n 



' NxN I ( n +N)x(n+N) 



is a diagonal matrix. Thus under (v), for any e > and any I > 0, there exist two positive 
constants v\ = min{l, aY/y} • (1 + y/Y) ^1 — jj^yj and vi = max{l, aY/y} • (1 + 
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y/Y) ^1 + y/ ~\ such that almost surely 



(1.3) lim Af' l+QnT ^ > z/i, lim A^ +a " T ^ < v 2 

p,n,iV— 5>oo p,n,N^>oo 

and 

(1.4) P ( A f" +a " Tjv < v x - e) = o{n~ l ) P (A^ +Q " Tjv > ^ + e) = o{n~ l ). 
One may refer to [1] for the proof of (|1 .3j) and (jl.4p . 

Remark 1.3. Under the assumptions (i) and (ii) in Theorem li.il it is proved that the 
ESDF of the sequence {S n } has a non-random limit which is known as the Marchenko- 
Pastur (M-P) distribution 13]. Yin [T7] and Silverstein [13] investigated the LSDF of 
the sequence {S n TV} assuming (i)-(iii) of Theorem \l.l[ IfTw is invertible, Bai et al [5] 
gave the LSDF of the sequence {S n T^ }. 

Remark 1.4. If min{y,Y} < 1, by (v) we know that at least one of the matrices S n and 
Ttv is invertible a.s.. Without loss of generality, we assume Y < 1. So Tjy is invertible 
a.s.. Then we have 

(1.5) B n = S n T^ (S n T^ + a n iy l , 

which is a function ofSnTj^ 1 . Via t = a n t/(l — t) we can recover Theorem 5.3 in [5] from 
our Theorem \1.1\ directly. Thus our Theorem \1.1\ includes Theorem 5.3 in [5] as a special 
case. 

For the purpose of multivariate inference, it is of interest to know the limiting distri- 
bution of these LSS (| 1 . 2 j) . Thus, we will give the central limit theorems (CLT) of LSS of 
Beta matrices. In order to present this result, we need more notation. Denote 

<B n (x)=p(F B "(x)-F (x)), 

where Fq is the limit distribution of F B ™ with a, y, Y replaced by a n , y n , YJv, respectively. 
For any function of bounded variation G on the real line, its Stieltjes transform is defined 
by 

s G (z) = j -JL-dG{\), z £ C + = {z E C : %z > 0}. 

Then we have: 

Theorem 1.5. In addition to the conditions (i)-(iv) in Theorem li.il we further assume 
that: 

(1) Exfj = Ex^ = t, E|xn| 4 = m. x , E|xn| 4 = m x and max{m x ,m x } < oo, where 
t = 0, when both X n and X^v are complex valued, and i = 1 if both real. 

(2) Let fx, . . . , fk be functions analytic on an open region containing the interval [q, c r ] 
where ci = i / 2 ^ 1 (l — ^/y) 2 , c r = 1 — av^ (1 — \/Y) 2 , and v 2 is defined in Remark 

Then, as mm(n, N,p) — > oo, the random vector 



(J fidK n {x)^j ,i = l,...,k 
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converges weakly to a Gaussian vector (Gf L , . . . ,Gf k ) with mean functions 
wn t z f(l-Yy^(z) + 2-s(z) + l-y 



AniJ JK a + z' 6 V (1 -Y)"s 2 + 2's(z) + 1 

+ h j hi ^rz )dlog & ~ Y ' k ' 2{z){1 + ' S ' {z)r ^ 



m T — t — 2 



yfi(^KHz) + l)- 3 d-s\z) 



2tri J a + z 

+ m *~ t ~ 2 S " Y"8 2 (z)(l + a (*))- 2 )dlog(l - Y-s\z)(l + -Hz))- 2 ) 

Am J a + z 

and covariance functions 

Coo{G fi ,G fi ) = ~^rff rozr^ 

y( TOa! -t-2)+y(m»-t-2) if / fi(^- 1 )f j (^)d-s(z 1 )ds(z 2 ) 



4vr 2 J/ (s(zi) + l) 2 (s(^) + l) 2 ' 

where 

s(z) = s F (z), s(z) = - — s(— ]— ) ^— > = - y) + ys(2;) 



y ( \ l-Y-z + ^/(z-l-Y) 2 -AY .. Y ! 

AZZ f/ie above contour integrals can be evaluated on any contour enclosing the interval 

r_ac|_ ac r 1 
Ll-c/' l-c r J- 

Remark 1.6. Actually, this result should be right under the condition that ft is analytic 
(or continuously differentiable) on an open region containing the interval [<i,i r ]. However 
its proof is more difficult at the current stage because we don't have the following results 
of Beta matrices: the exact separation of eigenvalues, the limit of the smallest and the 
largest eigenvalues and the convergence rate of the ESDF . 

Remark 1.7. IfY< 1, Zheng in [18] established the CLT of the LSS o/jSnT^ 1 } which 
is based on [3]. It is apparent that our Theorem \1.5\ covers Zheng's result. 

Remark 1.8. If {xij} and {x^} are independent standard normal random variables and 
p < minjra, N}, Beta matrices can be seen as Beta-Jacobi ensemble with some parameter 
(3. Some related results about this ensemble can be found in [9] and the references therein. 

This paper is organized as follows: In Section 2 we present the proof of Theorem 11,11 
Theorem 11.51 is proved in Section 3 and Section 4. Some technical lemmas are given in 
Section 5. 



2. Proof of Theorem 11.11 



In this section we will give the proof of Theorem 11.11 The main tool we use here is the 
Stieltjes transform. Its function can be explained by the following two lemmas. 



6 
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Lemma 2.1. (Lemma 1.1 in [6]J For any random matrix A n , let F An denote the ESDF of 
A n and Sj?a„ (z) its Stieltjes transform. Then, if F An is tight with probability one and for 
each z G C + , s F A n converges almost surely to a non-random limit sf(z) as n — >■ oo, then 
there exists a nonrandom probability distribution F taking sp(z) as its Stieltjes transform 
such that with probability one, as n — > oo, F An converges weakly to F . 

Lemma 2.2. (Theorem 2.1 in |15| ) Let G be a function of bounded variation and xq € 
R. Suppose that lim Qsg(z) exists. Its limit is denoted by Qsg(xo). Then G is 

differentiate at xq, and its derivative is it~ 1 Qsg(xo). 

Theorem 11.11 follows from the following Theorem 12.31 

Theorem 2.3. In addition to the conditions (i)-(ii) in Theorem \l.l\ we assume that: 

(1) . {T n } is a sequence of p x p Hermitian matrices with uniformly bounded spectral 

norm in n with probability one and the ESDFs of {T n } almost surely tend to a 
non-random limit H_. 

(2) . The smallest eigenvalue of matrices {Sn + anT^} almost surely tends to a positive 

value as n tends to infinity. 

Then we have F— n — ^» F_, where B n = S n (S n + ot n 'T_ N )~ 1 and F_ is a non-random 
distribution function whose Stieltjes transform s = s(z) = sf(z) satisfies 

K ! - J {l-z){l-y{l-z)(zs + l))-azt ~ y h 

and in the set {s:s£ C + } the solution to (|2.1[) is unique. 

By Lemma |2. II we know that to prove Theorem 12.31 we just need to prove three conclu- 
sions: (1) {F— «} is tight a.s.. (2) s p b„ s with s satisfying (|2.ip . (3) The solution to 
(|2.ip is unique in C + . Now we prove Theorem 12.31 step by step. 



2.1. Proof of Theorem 12.31 Step 1: Applying Lemma [5.21 directly, we have for any 
x,y > 

F^{(xy, oo)} < F s "{(x,oo)} +F (s " +Q "^) _1 {(y )00 )} 

(2.2) = F Sn {(x,oo)} + F Sn+an ^{(0,l/y)}. 

It is known that, under the assumptions of Theorem ll.il with probability one F Sn tends 
to the M-P distribution Fmp, which has a density function 

fzk^Vib ~ x )( x ~ a )' ifa<x<6, 
I otherwise, 

and has a point mass 1 — 1/ y at the origin if y > 1, where a = (1 — y^y) 2 and b = (1 + yjy) 2 . 
Thus {.F Sn } is tight almost surely. 

On the other hand, by the second assumption of Theorem II. 1\ the second term on the 
right hand side of (12. 2|) can be arbitrarily small as n is large, provided that 1/y is smaller 



(2-3) flJx) 
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than the smallest eigenvalue of the matrices {S n + a n T N }. Thus {F B "} is tight almost 
surely. 

Step 2: Recalling the definition of Stieltjes transform we have that for z £ C + 
(2.4) a F * n (z) = -J2-B = -tr(B n -zI) l . 

Here we have used the fact that B n has the same eigenvalues as 

S*' 2 (S n + a^)" 1 Sy 2 . 
Denote B e = S n (S n + ct n T N + with small e > 0. From Lemma 15.3} we have 

L 3 (F B„ _ F B e) < 1 tr(a _ _ 

n 

By the fact 

B n -B £ = eSy 2 (S n + a n T^)- 1/2 (S n + a n T^ + eI)" 1 (S n + a n T w )^ 1/2 Sy 2 
< e (S n + a n T N + el)' 1 
together with Condition (2) in Theorem 12.31 we obtain almost surely that 

L 3 (F— n — F— s ) < Ce 2 , 

which implies 

lim lim L(F^ — F— £ ) = 0. 

Next we consider the LSDF of Bg . Noticing that the matrix a n T_ N + el is invertible for 
any e > 0, we have 

B = I - ( B F + 



where B e = S„ (a n T N + el) 1 . Thus we get that 

F^(x)=F^(-^ 1) 

1 — x 

and 

11 z 

( 2 - 5 ) s fRs (z) = - h - yS f B E (: )• 

1 — z (1 — z) z P ~ E 1 — z 

Silverstein in |13j derived that for any z £ C + , the Stieltjes transform of the ESDF of B e 
has a non-random limit, denoted by sg(z), which satisfies the equation 

where H £ is the LSDF of (a n T N + el)" 1 . Note that 3(z/(l -z)) = \l- z\ 2r =sz > 0. Thus 
by (I2.5p we get that almost surely s f b e (z) tends to a non-random limit, denoted by s e (z), 
which satisfies 

(1 - zfs £ (z) - (1 - z) = [ — dH £ (t). 

J t(l-y- y(T^)((l - zfs^z) - (1 -z)))- jl- z 



.t- 1 
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By definition of and H_ , we have that 

dH £ {t) = -dH(- 
Therefore letting e — > we have 

' ~ J (l-z)(l-y(l-z)(zs + l))-azt ~ W 

Step 3: From Lemma 12. 11 we conclude that there exists a distribution function G with 
support C [0, 1] satisfying for any z G C + , 

(2.7) s(z) = I —^—dG(x). 

Jm G x - z 

Noticing that $5z(a + z)" 1 = \a + zl -1 ^ > 0, we infer from (|2.7|) that 

-dG(x) 



(a + z) 2 \a + zj a + z (a + z) 2 J^ G x - a + z 

= I ~~7T~ = /°° (-^ . 

j^, G ax — z(l — x) J x — z \a + xj 

Thus 

( 2 - 8 ) * = -CO = rrr^ f-rr) - ~r- 

(a + z) z \a + z/ a + z 

is a Stieltjes transform of the distribution function G(-^-^) with x G [0,oo). Notice that 
even if has a point mass at x = 1, we have ^^^j = 0. Thus (|2.6[) can be 

represented as 



i(l - y - yzs{z)) - z y ~ y t !h 



where M + = {t : t G M, i > 0}. It is shown that the solution of the last equation is unique 
in C + (see [13]). Thus we obtain that (|2.6p has a unique solution in C + , which completes 
the proof of Theorem 12.31 



2.2. Proof of Theorem 11.11 Using Theorem 12.31 and Remark 11.21 we know that the 
Stieltjes transform of F is the unique solution in C + to the equation 

(l-y(l-z)(zs + l)) + at 
(1 - z)(l - 7/(1 - z)(zs + 1)) - azt 



(2-9) - = / , t ^^/^..^^^I/^^^.^ ^CO- 



Here F^ p is the limit of F Tn which is also the M-P distribution. After some calculations 
we may represent the last equation as 

w f 1 , _,y 



( 2 - 10 ) S = — z -^ 2 i I d- Z W dE ^ 



az 



where w = 1 — 7/(1 — z)(zs + 1). Recalling (j2.8|> . we have that 

1 a az s 

s ( 2; ) = 1 1 + 7i Iv? s ( 



1 - z (1 - z) 2 v l - z' 
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and 

w = 1 - y{l - z)(zs + 1) = 1 - y - 

which implies 

(1 - z)w (1 - z)(l - y) az 

= ys{z )• 

az az 1 — z 

Noticing that 9^ > 0, we have < and 

° I — z 7 az 



[ I dF Y (t) = s y 

where is the Stieltjes transform of the M-P distribution F^ p . Since 



s Y (z) 

a mp \"J 

the equation f|2. lOj) implies 



1-Y-Z + ^/(z-l-Y) 2 - AY 



1 vo l-Y-^ + J(^-l-Y)i-4Y 



{1-z)m // (l-z)ro 
az ' Y ^ az 

Z az 2 I 



where, and throughout this section, the square-root of a complex number is specified as 
the one with positive imaginary part. The solution to this equation is 



= (1 + y)(l-z)- az(l -Y) + ^((1 - y)(l - z) + az{\ - Y)) 2 - Aaz(\ - z) _ 1 
S[Z) 2z(l - z)(y{\ - z) + azY) z 

Now using Lemma 12.21 and letting z \. x + iO, n $ss(z) tends to the density function of 
the LSDF of B n . Thus the density function of the LSDF of B„ is 



( ^igrj^^^ ^. ^ 4ax(l - x) - ((1 - y)(l - *) + ax(l - F)) 2 > 
[ 0, otherwise. 
Or equivalently, 



f > /((a(l-y)-l+y) 2 +4 a )( 3!r - a! )( 3! - 3! ,) ., . 

27rx(l-x)(y(l-a:)+axY) ' 11 X ' ^ X ^ Xj- ' 

[ 0, otherwise, 
where xi,x r = ^ 2a "( 1 "^)|^( 1 ~^)~ 1 +^T^QV?/-i/^+^ \ _ n w we determine the possible atom 
at and 1. When z — > with ^sz > 0, we have 

3f[((l - y)(l - z) + az(l - y)) 2 - 4az(l - z)} 

= 29z{ [(1 - Y)a - 1 + y) [(1 - y)(l - Uz) + a(l - KJSfte] - 2a(l - 2Kz)} < 0. 



By the fact that the real part of \/ g(z) has the same sign as that of the imaginary part 
of g(z), we obtain that ^((1 - y)(l - z) + az(l - Y)) 2 - Aaz(l - z) < 0. Thus 

V((l -y){l-z) + az(l - Y)) 2 - Aaz(l - z) -> -|l-y|. 



10 
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Consequently, 



F{0} = - lim zs(z) = B — ^ — - + 1 

z^o 2y 



if y > 1; 



0, otherwise. 
When z —> 1 with 9z > 0, we have 

3[((1 - y)(l -z) + az(l - Y)f - Aaz(l - z)\ 
= 23z{ [(1 -Y)a-l + y) [(1 - y)(l - Sfcz) + a(l - Y)$tz] - 2a(l - 25ftz)j > 0. 

Hence we get &y/((l - y)(l - z) + az(l - Y)) 2 - 4az(l - z) > 0. Thus 
V((l - y)(l - ^) + az(l - r)) 2 - 4az(l ^z) ->■ a|l - Y\. 

Consequently, 

F { i) = - liznt* - iw,) = n-y|-(i-n = / ^, if »- > i; 

1 1 z->V ' w 2y \ 0, otherwise. 

Then the proof of Theorem 11.11 is complete. 

3. Framework of proving Theorem 11.51 

In this section we will give the proof of Theorem 11.51 Recall the definition of the 
Stieltjes transform of a distribution function G(x). Now we extend the Stieltjes transform 
to the whole complex plane except the interval [Q,c r ] analytically. Since every fk(x) is 
analytic on an open region containing the interval [q,c r ], we assume that the analytic 
region contains the contour C = {z G C : ffiz G [q — 6, c r + 9], 3\z = ±0} \J{z G C : 5Rz G 
{q — 9,c r + 9}, Qz G [—9, 9]}. Here 9 can be small enough. By Cauchy's integral formula 

h{x) = — f dz, 

ZTTl J c Z — X 

we have for / > 1 and complex constants a±, ■ ■ ■ ,ai, 

(3.1) YjL k p ^ h(x)dF n {x) - f f k (x)dF (x)^j = <f e fk(z)S n (z)dz, 

where S n {z) = p{s n {z) — sq(z)) and sq(z) is the Stieltjes transform of Fq with constants 
y and Y replaced by y n = p/n and Y n = p/N. We remind the readers to notice that the 
above equality may not be correct when some eigenvalues of B n fall outside the contour. 
However, by the exact separation theorem in [4], we know for y > l(or Y > 1) and large 
enough n(or N), the mass at the origin (one) of F n will coincide exactly with that of -Fo 
and with overwhelming probability all the other eigenvalues of B n fall in [q — 9,c r + 9}. 
Thus to prove Theorem 11.51 it suffices for us to derive the limiting distribution of S n (z). 
Write 

S n (z) = p(s n (z) - s N0 (z)) + p(s m (z) - s (z)) := S nl + S n2 , 
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where sno(z) is the unique root of the equation 

(1 - y n (l - z)(zs N0 (z) + 1)) + a n t 



sno 



-dF TN (t) 



(1 - z)(l - y n (l - z)(zs N0 + 1)) - a n zt 
in the set {s m (z) G C + }. Using the notation s N0 = s m (z) = ( a ^ z y2 S N0 (^f^) - 
s = s (z) = {a ^ z) 2 S (-^f^) - ^pj, s N0 (z) = -z- x {l - y n ) + y n s N o(z) and s (z) = 
- y n ) + y n s (z) we have 

1 fdF T »(t) 1 /-di^(t) 



s'ato 7 i + SAT s J t + s 

The difference of the above two identities yields that 

so ~ sno _ f (sp - s N0 )dF T » (t) f dF T " (t) - dF^N (t) 



S0S N0 J (t + SNo){t + So) J t + S 

Then we get 

,„ , f dF*»(t)-dF%(t) ( .... f dF^(t) 

(3.2) S - SATO = VnSoSNO / — — : " 1 ~ VnSQSNO ' 



t + S V J (t + SNo)(t + s ). 

Let s Tn be the Stieltjes transforms of F N and then from Lemma 1.1 in [3] (or (6.32) in 



18] ) we have the conclusion that 

dF T »(t)-dF%(t) 



P 



^=p(^Ho)-^Ho)) 

t + s 



converges weakly to a Gaussian process <3?i on C with mean function 



{l-rs(i)/[l + !(i)|'( ! ' 1 '1-Y[s(z)f/[1 + -s(z)Y 

and covariance function 

CHz 1 )ycs( Z2 )y i 
[• s -(^i) - * (* 2 )] 2 (^i)-^ 2 )) 2 
y(*(*0)'(«te))' 



C<w($i(zi),$i(« 2 ) = (t+1) 

(3 - 4) + ^- i - 2 \i + -s( Zl)? [i + -s( Z2) r 

where a (*) = s^ p (-s(z)), s^ p (z) = -z~ l (l-Y)+Y s^z) and (a (*•))' = ss'mp(*0U=-s(*i)> 
? = 1,2. Using the identity 

1 /"^(t) 
(3-5) = ~i r + y ' 



s(z) J t + s(z) 
one can get that 

s (z)s N o(z) 



1 - y n so(z)s NO (z) J (t+ s No(z l)(t+s (z)) dFTN ( t ) 

converges to 

(3 ' 6) 1-^ 2 (.)|(2W^) = Tz" {z) '= *' {Z) - 
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Thus from Lemma 1.1 in [3], (|3.2H and the above arguments we obtain that {S n2 (-)} 
forms a tight sequence on C and S n2 {-^^) converges weakly to a Gaussian process — (a + 
z) 2 s'(z)&i(z) with mean function 

E(-(l + zfs'iz^iz)) = -(a + zfs'{z) ■ (pT3|) 

and covariance function 

Cew(-(a + zi) 2 s'(zi)^>i(zi), -(a + z 2 ) 2 s'(^)^i(^)) 
= (a + zi) 2 (a + z 2 ) 2 s'(z x )s'{z 2 ) ■ ([33]). 

Recall the notation W = 1 — y(l — z)(zs + 1) and suppose we have the following lemma: 

Lemma 3.1. Under the conditions of Theorem \1.1\ and z 6 C, we have that given Ttv = 
{all T^}, {S n i(-)} forms a tight sequence on C and S n i(z) converges weakly to a two- 
dimensional Gaussian process $2 ( z ) satisfying 

E(* a (*)|I*r)^ J mpU 



, Q _, . / , 0W1 v 3/ (l-*)i- 2< rf dF mp(*)/ ((i_ z )^_ zcrt )a dF mp(*) 

(3.7) +(m a; -t-2)(l -z)ya7 d (1 _ z)2 ^ 2 ' v ' 

and 



\ J ((1 — zijro— ziat)((l — Z2)ro— 22«t) Tip 1 - / 



(3.8) +(». - t - 2), / ('-?>"W <<( t) f (izi2Mf2L dF K (t) 



(1 — — z\at mp J (1 — 2:2)117 — Z2"^ 



We postpone the proof of this lemma to the next section. Now we use the notation 

s = s( z ) = j^v s (^) - and s ( z ) = - 0) + i/*(*0 to § et 

/ _ _ \ / \ GiZ , GLZ , 

(3.9) = -— — S (t— ), 

1 J2- X Z 



which can be used to rewrite (|3.7p and ( 13.811 as 
(3.10) E($ 2 (— ^- )|Tjv) ->t 



* mo- s + z ) 2 / at (*(*)) 3 (*(*)+*) 3 ^Ip(*) 



a + Z ' ' (l - y J (s(z)) 2 (s(z) + t)~ 2 dFl p (tj) 

(3.11) 

+(mj; - t- 2)y(a + 2 



2 / S(*) + ty 1 dFl p {t) J at (s(z)) 2 (s(z) + t)~ 2 dFl p {t) 



l-yf(s(z)y(s(z)+t)- z dFy p (t) 
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and 



s'(z 1 )s'(z 2 ) 1 



Coi;(*2(— ^— ), $ 2 (— i-)^) -+ (t+ l)(a + zi) 2 (a + z 2 ) 2 , 
(3.12) 

+(m x -i-2)y{a + z l ) 2 {a + z 2 ) 2 J ats 1 (zi) (s(zi) + t)~ 2 dF^ p (t) J ats' \z 2 ) (s(z) + t)" 2 dF* p (t). 
Here we used the fact that (similar to (|3.2|) ) 

Zl " Z2 = " ^ / ' (Zl) ' (Z2) (S(zi) + ( ' (Z2) + ^ dF ™v [t)) 

As the mean and covariance of the limiting distribution are independent of the condi- 
tioning Tat, we conclude that <S n i and S n2 are asymptotically independent. Then from 
the above argument we can get that S n (j^) converges weakly to a Gaussian process 
$3(2) = -(1 + z) 2 s'(z)®i(z) + $2(1^) with mean function 



a{l-Y-s(z)/[l + -s(z)} 2 } 2 
-^-i-2)(a + z) 2 s'(z) n-Wl-H + ^W]- 



a{l-Y[s(z)] 2 [l + 's(z)}- 2 } 
y{a + z) 2 J at (s(z)) 3 (s(z) + t)~ 3 dF% p (t) 

l-yf(s(z)) 2 (s(z)+t)- 2 dFy p (t)y 



^ 2 / s(z) (s(z) + ty 1 dFl p (t) J at (s(z)) 2 (s(z) + t)~ 2 dF% p (t) 

+(m x - t - 2)y(a + z) ^— s 5 — 

l-yJ(s(z)) 2 (s(z) + t)- 2 dFY p (t) 

and covariance fucntion 

Cov(^ 3 (z 1 ),^ 3 (z 2 )) 



= (a + Zl ) 2 (a + z 2 ) 2 s'{ Zl )s'(z 2 ){t + 1) f || 
+(1 + Zi) 2 (l + z 2 ) 2 s / (z 1 )s'(z 2 )(m x - t - 2) 



(zi) - 's (z 2 )] 



2/1 , ^2,//^,//^/^ on ^(^oyc*^))' 



[l+a(^)] 2 [l+ s(%)] 2 
(t + l)(a + zi) 2 (q + z 2 ) 2 
(*i - ^) 2 

+(m x -t-2)y(a + Zl ) 2 {a + z 2 ) 2 J ats' \ Zl ) (s( Zl ) + t)~ 2 dF^ p {t) J ats'{z 2 ) (s{z) + t)~ 2 dF^ p (t). 
which together with ()3.ip and Lemma |5. II implies Theorem 1 1.5 1 

4. Proof of Lemma I3TTI 

In this section we give the proof of Lemma 13.11 Following the similar truncation steps 
in [3] we may truncate and re-normalize the random variables {xy} as follows: 

\xij\ < 5 n \/n, Kxij = and E|xjJ 2 = 1. 
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Here 5 n — > which can be arbitrarily slow. Based on this truncation, we can verify that: 
(4.1) E|xi/ = m x + o(l), 

and if X n is complex valued, 

Exfj = 0(n _1 ). 

We will introduce some notation and provide some bounds in the first part of this section. 
The proof of Lemma 13.11 will be given in the next part. The main procedures of the 
proofs, including the Stieltjes transform, the martingale decomposition and Burkholder's 
inequality, are routine in RMT, hence we will outline them without detailed descriptions. 
Interested readers are referred to Bai and Silverstein [1]. Throughout the rest of the paper, 
constants appearing in inequalities are represented by C which are nonrandom and may 
take different values from one appearance to another. 

4.1. Definitions and some basic results. In this part we introduce some notation and 
some useful results. Firstly we assume z = u + i9 with 6 > 0. For simplicity, write S = S n 
and B = B n . Let D = D(z) = B - zl, F = F(z) = (1 - z)S - za n T N and I be the 
identity matrix. Define = n _1 / 2 X(.j) where is the i-th column of X n , Sj = S — r^r*, 
Bi = Si(Sj + a n T N y\ D, = T>i(z) = B, — zl and F< = F t (z) = (1 - z)Si - za n T N . Let 
Ej = E(-|TjV) r i> • • • j r i) and Eo = E(-|Tjv)- Moreover introduce 

Wi = Wi(z) = - — — j- , wf = wj r (z) 



l + (l-z)r*Fr 1 (z)r i ' 1 * w 1 + n -i(i - z )trFr 1 ( z ) ' 

„K _ E/ \ _ \ 

1 i[) l + n-i(l-z)E irF- 1 (z)' 

7i = 7i(z) = rjFr 1 ^ - n^EotrFT 1 , Vi = Vi (z) = rjFr 1 ^ - n^rFr 1 , 

6 = &{z) = n-HrFr 1 - n^EotrFr 1 , 

s n = s n (z) = s F B n (z), s = s(z) = s Fv ,h(z), s = s (z) = S F y n ,H n (z). 

Obviously we have, 

(4.2) 7i(*) + &(*)■ 

(4.3) roi = tuf - (1 - z)wfw ai = zof - (1 - z)(wf)\ + (1 - z) 2 (rof )V 7 2 , 
and 

(4.4) wi = wf - (1 - zjrofo?*^ = wf - (1 - z)(rof )% + (1 - z) 2 (tuj r fw^f. 
It is easy to verify that 

(4.5) 9(1 - z)- 1 = 0\1 - z\~ 2 
and 

(4-6) 9rjFri( z ) ri = 0r * F -i( z ) ( Si + « n TV) Fr 1 ^ 

have the same sign. Therefore from the definition of Wi, we have 



(4.7) |^| 



1 1 



l-z-r- + r*Fy 1 (z)r i 



II 

< J— 
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\l-z\ 



11 - z\ 



Similarly we can obtain 

(4-8) \wt\<'—^, |tuf|< y 

By the fact that 

(4-9) IIF^WII = \\V;Hz)(Si + anT^)" 1 !! < C6~ x 

and Lemma 15.41 we have for any I > 2 

(4.10) 



r<x2i-4 



In the last inequality we used \xij\ < 6 n y/n. For any invertible matrices M, M + rjr* and 
N, using 

(4.11) r^M + rir*)" 1 = r?M _1 , M" 1 - N _1 = — N _1 (M - N)M _1 , 

v ) i\ i) l + r *Mri * v ; = 

we obtain that 

(4.12) F- 1 ^) - ¥-\z) = -(1 - z)w i Y^T i xrF^, 

which together with (|4.6p - (|4.9p implies that for any Hermitian matrix M with ||M|| < C, 

(4.13) |trF _1 (^)M - trPr^^Ml = |(1 - z)w l r*Fr 1 MF- 1 r i \ < CO' 1 . 

Lemma 4.1. Under the conditions of Theorem \1.5l we have for any non-random Hermit- 
ian matrix M with ||M|| < C and I > 2, 



C a 

^\n- 1 ti¥' 1 {z)m-n' 1 ¥ J QtYY- 1 (z)M\ l < 1 n 



21-4 



n i/2+i ffi V where z = u + i9. 
Proof. The martingale decomposition (one can refer to [1] for more details) gives 



trF _1 M - E trF _1 M = ^(E; - Ei_i)tr(F -1 M - F ( -1 M) 

i=i 

n 

= -(l-z) J2(Ei - Ei_ 1 )roirtFr 1 MFr 1 r i 



i=l 



-(1 - z) ^(E 4 - E^m? T^MFr^i + (1 - zf ^(E, - E^mfw^Y^MF^. 



i=i 



Here we used (|4.12p and (|4.4p . From (|4.9p and Lemma 15.41 we obtain that 

C5l l ~ A 



E|r*F^ 1 MF,~ 1 r i — n~ 1 trF," 1 MF~ 1 | i < 



n 



Q2l 



Thus it follows from 



and Lemma 15.61 that 
E 



E 



£(Ei - E^O^fr^F-^F- 1 ^ 

i=i 

^(E, - E^Vf (r^MF^n - n'HrF^MF^) 



< 



Cn l ' 2 5l l - A 



n 



031 
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On the other hand, from (|4.8j) . (14.130 . H4. 10H and Lemma 15.61 we also have 



E 



' Cn l / 2 5 2 n 1 - 4 
< - 



n 



031 



i=l 

which completes the proof. □ 
Remark 4.2. From the last lemma and (|4.13p . one can easily verify that for any I > 2, 

fi l/2?2l-4 

(4.14) E\trF- 1 (z)M - EtrFr 1 (z)M|' < — — ^ — . 

Furthermore, by combining (|4,2p . (|4.10p and (|4.14p with M = I, we have for any I > 2, 

(4.15) Ehi\ l < 1 n . 

n 

Denote Sjj = S — r^r* — rjr* for i ^ j. Correspondingly, let By = Sjj(Sjj + a n Tzv) , 
Dy = Djj(z) = By — zl, Fij = Fij(z) = (1 — z)Sy — zaT^ and assume ||(Sy + 
a n TV) -1 || < oo. Moreover we have 

™ij = ™ij(z) = -—J, w^-l/ \ i w tj = w ij( z ) 



l + (l-z)r*Fr. 1 (z)r/ v " l] I + n~\l - z)trFjf 

CT i7 = W ii\ Z ) 



- l+n- 1 (l-z)E tvFr j 1 (zy 



7ij = 7ij (z) = v*F^{ z ) Tj - ,^ 1 E trFr. 1 (z), % - = mj {z) = r) F,, 1 ( : )r ; - n^rF". 1 ^), 

Zij = = rThvFr l (z) - n- l E ^F- l {z). 

We can get the same bound as we did in (|4.2|) - (|4.13|) by changing the subscript i to ij. 
Thus from now on when we consider these bounds we will ignore the subscripts. Let 
H12 = Hi2(z) = (1 — z)' n ^-vo\ 1 \. — za n Tw- We have the following lemma. 

Lemma 4.3. Under the conditions of Theorem \1.5\ and z = u + id, we have for any 
1 < k < p, 1 < i < n and non-random matrix M with ||M|| < C 

(4.16) E e* fc F- 1 (z)Me fc = e* k U^(z)Me k + 0(n 



-l/2> 



where e^ is the p- dimensional vector with the k-th coordinate being 1 and the remaining 
being zero. 



Proof. Using (|4.1ip we can check that 

G7?i(l 

n 



FT\z) = U^(z) + ?M_zA £H£te) (f^) - F-.^z) 



(4.17) =Hr 2 1 (^+%)-^( 2 )-%), 
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where 



H {2) =(1 - z)wf 2 ~ n- 1 U^ 1 (z)F- j L (z] 



H {3) =(1 - z) - uj^Kii&rtfF^iz). 

Note that, similar to (|4.5p . either the real parts or the imaginary parts of (1 — z)wf 2 an d 
— z have the same sign. Thus we have for any t > 



n-1 je 



(1 — 2) w 12 — za n t 

n 



< 



C_ 



\n- l2 \z)\\ < 



C 



(4.18) 

which implies 

(4-19) ,i"i,v~,„-^ 3 
Then it follows from (jIUl) . (I4.19P and Lemma El that 

< Cn- 2 elU u \z)H u \z)e k Eoe* k MF- j 1 (z)F- j 1 (z)M*e k + n- 2 M \e* k H^ (z)F^ ( z )Me k 
From (|4.19p we have 

(4.20) \efH~ 2 \z)e k \ and ^H^H^e* 

are both bounded from above. In addition, by (|4.9j) we get that 
(4.21) 
and 

(4.22) \etH^(z)F- j 1 (z)Me k \ < C6~\ 

Thus combining (gSJ, (l4"7L5j) . (gJJlD, (l43T|) . (1021) and Holder's inequality we obtain 

E |#(i)| = O(n^) and E \H {3) \ = 0{n~ l l 2 ).. 

Apparently we have EqH^ = 0. Thus the proof of the lemma is complete. □ 

Lemma 4.4. Under the conditions of Theorem 1 1.5\ and z = u + iO, we have for any 
1 < k < p, 1 < j < n and non-random matrix M with ||M|| < C 



e^MF^zjF^zjMe, < C\\F^(z)\\ 2 < CO 



(4.23) 

and 

(4.24) 



K\e* k F~ 1 (z)Me k - E e^F- 1 (z)Me fe | 2 = 0{n~ l ) 
Ele^F-^^Mefc - E Q e\F^ {z)We k \ 2 = 0(0. 
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Proof. Similarly to the proof of Lemma 14.11 we obtain that 



e^F- 1 (z)Me fc -E e^F- i (z)Me fe 

n 

-(1-z) ^(Ei - Ei_i)wf r^ekelF^Mn 



i=l 



+(l - zf ^(E 4 - E^wf wwrjF^efcejF^Mri. 



i=l 



By Lemma 15.41 and (|4.21|) we have that 

ElrjpriefcejFr 1 ^ - n~ l trF~ l e k e%F- l M.\ 2 < §E (e* k F^F^e k ) 2 < §. 



n- 



Thus from 
(4.25) 



and Lemma 15.61 we have 



E 



j=i 



< 



C 



n 



Also from (|4.8p . (|4.13p . (|4.10p and Lemma 15.61 we can obtain 



E 



^(E 4 - E l . 1 )wj r w l 7 li r*F~ 1 e k elFr 1 Mr i 



i=l 



have the same bound as (|4.25p . and then we get (|4,23p . Applying (|4.12p and (|4.2ip we 
can obtain (|4.24p directly. Therefore the proof of this lemma is complete. □ 

Lemma 4.5. For any non-random matrix M with ||M|| < C and z\ = u± + iO±,Z2 = 
U2 + i02 with minj^i, 62} > 0, we have 



1 



(4.26) E —trMF^ 1 (zi)E i (F~ 1 (z 2 ) ) — E ( —trMF~ 1 (zi)E i (F^ 1 (z 2 ) ) ) = 0(n 



7? 



1 



Remark 4.6. Checking the proof of Lemma \4-5{ we see that Lemma \4-5\ holds as well 
when we replace Ej(F^ 1 (z2)) by F~ 1 (z2)- The main difference in the arguments is that we 
do not distinguish between the cases j <i and j > i when dealing with the latter. 

Proof. Using the martingale decomposition, we have 

hrMF-\zi)E i {F-\z2))-E (^trMF-\zi)E i {F-\z2)) 



- ]T(E, - E,_0 [trMFr\z 1 )E i (Fr\ Z2 )) + irMF"- 1 (z\ )Ej (F^ 1 (22) ) 



1 n 

_^( Ej -E i _ 1 )(/C 1 +/C 2 + /C 3 ), 
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where (via (|3~T2|) ) 

/C 2 = -^(z 1 )r*F. 1 (z 1 )E ! (F. 1 ( Z2 ))MF^ 1 (2i)r J 
/C 3 = -trMF^( Zl )E l ^w lJ (z 2 )F^\z 2 )r j r*F- j 1 (z 2 )). 
Note that by KW\ 

(4-27) K-lll'i^WII 2 = l^rJFr^^Fr.Hz)^! < C, 

which implies that /Ci is bounded. 
When j > i, applying (j4.3|) to get 

(Ej - Ej-O/Ci = (Ej - E J _i)wf 2 (z 1 )(/Cii - /Ci 2 ), 

where /Ci 2 = 7fcj(^i)/Ci and 

/Cn = r*Fr. 1 (zi)E i (^(z 2 )G fc (z 2 ))MFr. 1 (z 1 ) rj 

-n-HrF- 1 (21 )Ej (o7 - (z 2 ) G y (* 2 )) MFr. 1 ( Zl ) 



with Gij(z 2 ) = Fr. 1 (z 2 )r i r|Fr 1 (^ 2 ). We conclude from (jOTj) . (Oil . (|Q5j> . LemmaE 
Lemma I5U1 and ||M|| < C that 

1 n m n n 

E|- V(E, - E^xX/Cn - /C 12 )| 2 < ^ V (E|/C n | 2 + E|/C 12 | 2 ) < -j. 

j>i j>i 

On the other hand, when j < i, we define Fj j 1 (z),w i j(z) and 7. .(z) using n, . . ., r,-_i, 

Lj+i, Li-i, Li+i, • • • , I„ as F~ j 1 (z),zu ij (z) and 1kj(z) are defined using ri, . . . , r,-_i, 
rj + i, . . . , rj_i, rj + i, . . . , r n . Here r 1} . . . ,r n are i.i.d. copies of ri and independent of 
{Tj,3 = 1, • • • ,n}. Let 

Kij^zuzz) = v*F- l {z 1 )F- 1 {z 2 )v, J , K lj2 ( Zl ,z 2 ) = r*F7 j \z 2 )MFr j \z 1 )r j . 

Applying the equality for w_kj(z 2 ) similar to f|4.3j) yields 

(Ej -Ej_i)/Ci = (Ej -Ej_i) Wijiz^w^^TZijiizi, z 2 )TZ ij2 (zi, z 2 ) 

= (Ej - Ej_i)(/Ci3 + tCu - - K-w), 

where 

= ■^ij{zi)w ij (z 2 )Tiji{z 1 ,z 2 )n ij2 (zi,z 2 ), 
/C14 = w i j(z 1 )^ ij (z 2 )n' 1 trF~ j 1 (z 1 )FT j 1 (z 2 )Tij 2 (z 1 ,z 2 ), 

/C 16 = tufa (^i)^i (*i )7ii (*i )Si (^)n" 2 trFr. 1 ( Zl ^ (z 2 )trF_r l (z 2 )MF^ [z x ) 

with 

Tiji(*i,Z2) = K a - n- 1 trFT. 1 (z 1 )FT j 1 {z 2 ), T ij2 (zi,z 2 ) = TZ i2 - n- l trFr l (z 2 )MFr l ( Zl ). 
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Apparently F_^ l (z),w i j(z) and 7 . (z) have the same bound as Fr. (z),mij(z) and ^ij{z) 
respectively. Thus it follows from Lemma 15.41 (|4.9p and (|4.27p that 

(4.28) E|7Jji(*i, z 2 )\ 2 < -, E\Ti j2 (zx, z 2 )\ 2 < - 

n n 

and 

(4.29) n- l \tr¥^{ Zl )YT^ Zl )\ < C , n^tr^^MF^^)] < C. 

Therefore combining ([4l5]) . (14~?7|) . (Ojl . (|428l) and (!Q9j) . we can obtain that for t = 
3,4,5,6, 

E|(E i -E J -_i)X: lt | 2 = 0(n- 1 ). 
This via Lemma 15.61 implies that 



n 

E|- J^(Ej -Ej_i)/Ci| 2 = 0(n" 2 ). 



n 

The terms /C2 and /C3 can be similarly proved to have the same order. Then the proof 
of Lemma 14.51 is complete. □ 

Now we use (|4.17p to write that 

^-trMFr 1 (z 1 )E l Fr\ Z2 ) = hrMH^ 2 \z 1 )E i Fr 1 (z 2 ) 

(4.30) hrMH^izi^Fr^zz) - hrmH {2) {z 1 )E i F- x {z 2 ) - hTMH {3) (z 1 )E i Fr 1 (z 2 ). 
Then we have the following lemmas. 

Lemma 4.7. For any non-random matrix M with ||M|| < C and Z\ = u± + i9\,z 2 = 
u 2 + i0 2 with min{#i, 9 2 } > 0, we have 

(4.31) lEotrMi^^E^Fr 1 ^))! = O p (l), 
and 

(4.32) |E trMii' ( 3 ) (z 1 )E i (Fr 1 (z 2 ))| = O p (l). 
Proof. By (|4.12[) . we obtain that 

trMfl-^C^OEiCFr 1 ^)) 
= ^(^K 1 -^) ^ CTij .(, 1 )r*Fr.i(^ 1 )E i (Fri( Z2 ))MH7 2 1 (z 1 )Fr.i( Zl )r j . 

As ff^ (2) F~ 1 (z), F^ 1 (z), rof 2 (-z) and wi 2 (z) are all bounded when 9z > 0, we can get 
directly that for j > i, 

|E trMF (1) (z 1 )E i (Fr 1 (z 2 ))| < C. 
When j < i, note that we also have 

|E trMi/ (1) (z 1 )E i (Fr. 1 (z 2 ))| < C. 
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Then from (I4.12p . Ela^-I < oo and the definition of FZ (z), w_ij{z) and J--( Z ) i n Lemma 
14.51 we have 

^ow^z^F^izMF- 1 ^) - F j r. 1 (z 2 ))MHr 2 1 (^i)F-. 1 (z 1 )r i | 

= ^tuyC^^C^^ = 0(1), 

which completes the proof of (|4.3ip . 

Now consider (|4.32p . When j < i, using (|4.3p we rewrite the left hand side of (|4.32p as 

(4.33) = (1 - Zi)tuf 2 (zi)E ^ w ij( z lhij( z l)Tij3(.Zl, z 2 ) 

(4.34) +(1 - ^^^(zOEo?!- 1 ^^ J (z 1 ) 74J (z 1 )trF->(z 1 )E i (F- 1 (z 2 ))MH^ 1 (z 1 ) 
where 

T ij3 (z llZ2 ) = r*F-. 1 (z 1 )E i (F- 1 (z 2 ))MHr 2 1 (zi)r J - n" VFr.^i^F-^^MH^i). 

From Lemma 15.41 we have E|7Ij3(zi, z 2 )\ 2 = (9(n _1 ) which together with (|4.15p and 
Holder's inequality implies 

(Q3l) = O(l). 
For (|4,34p . we apply fj4.3|) again and obtain that 

IgUDI = | [(l-z 1 )zuf 2 (z 1 )YE n~ 1 Y,WiMhI j ^^ 

Here we have used the fact that |n _1 trF~ 1 (zi)Ej(F^" 1 (z 2 ))MH^ 2 1 (2i)| is bounded. Thus 
from (|4.15p we get that 

(OiD = om. 

On the other hand, when j > i, the above argument apparently also works if we replace 
Ej(F J " 1 (z 2 )) with Ej(F~ 1 (z 2 )). And the remaining term can be expressed as 

(l-^f^OE^^O*^ 

(4.35) = (1 - ^K^E^T^^t^ 

+(1 - z^wf^z^En- 1 ^ro ii (zi)ro ii (z 2 )7i i (zi)trF-^(zi)F-. 1 (z 2 )7I j2 (zi,z 2 ) 
-(1 - zi)(l - z 2 )wf 2 (z 1 )wf 2 (z 2 )En^ 2 ^w ij (zi)w ij (z 2 )^i j (z 1 )^ ij (z 2 ) 
■tiF^\z 1 )F^(z 2 )trFT j 1 (z 2 )MIl^(z 1 ) 
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(4.36) -((1 - z 1 )ruf 2 (z 1 )) 2 En- 2 J2w ij (z 1 )w i] (z 2 ) ( 7 ^i)) 2 

•trFr^zOFr^z^trF-.^z^MHr^^i). 

By Lemma 15.41 and a similar argument in (|4.3ip we can show that (|4.35p - (|4.36p ar e all 
bounded. Then the proof of Lemma 14.71 is complete. □ 

Lemma 4.8. For non-random matrix M with ||M|| < C and z\ = u\ + i6\, z 2 = u 2 + i0 2 
with min{#i,#2} > 0, we have 

E n- 1 trMH {2) (z 1 )E i (Fr l ( Z2 )) 

(4.37) = -(» - l)n" 2 (l - Zl )(l - z 2 )vjf 2 (z 1 )wf 2 (z 2 ) 

• trHr 2 1 (z 2 )MH^ 1 (zi)E trF,r 1 (zi)Fr 1 (z 2 ) + O p (l), 

Proof. It follows from (|4.12p that 

E trMi/ (2) (z 1 )E i (Fr 1 (z 2 )) 

= -(1 - Zl )(l - z 2 )wf 2 ( Zl ) Y^EoW.i j (z 2 )r*F^\z)F^\z 2 )r j r*F^ 
+(1 - Zl ){\ - 2 2 )^ 1 )^Eo^(2 2 )r^ 

j<i 

(4.38) 

= -(1 - Zl )(l - z 2 )zuf 2 ( Zl ) E 0Sj (z 2 )r;F. 1 (z)F7 1 (z 2 )r J T*F. 1 (. 2 )MHr 2 1 ( 2l )r J + O p (l). 

j<i 

In the last equation we used (|4.8p . (|4.9j) . (|4.19p and E|xjj| < C. Applying (|4.3p to rewrite 
the first term of (|4.38p as 

(4.39) -(l-^Xl-^K^K,^^ 

j<i 

(4.40) -(1 - zi)(l - 22)^(^)^2(22) ^E 7^i(zi, z 2 )T m (zi, *2) 

j<i 

(4.41) 

+(1 - zi)(l - z 2 )tu? 2 (;^ 2 (z 2 )£]E^ 

j<i 

where 7^4(21,22) = r *jlLij 1 ( z 2)~M.~lI^ 2 (zi)rj — n~ 1 trF_Jj 1 (z 2 )'Mli^ 2 (zi). The arguments in 
(l4T35|) - (06l) and (ET28D ensure that 

$M = O p (1) and gaU = O p (l). 

In addtion, from (|4.12p and (|4.13p . we have 

E trF-. 1 (22)H7 2 1 (2 1 ) = E trFr 1 (22)Hr 2 1 (^i) + O p (l) 
Then using (|4.17p again and repeating the arguments in Lemma 14.71 we obtain that 

EotrF- 1 (2 2 )Hr 2 1 (^i) = trH^H^H^i) + O p (l). 
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Combining the above arguments we conclude that 

E n- 1 trMF (2) (z 1 )E i (F7 1 ( Z2 )) 
= -(» - l)n~ 2 (l - zx)(l - z 2 )rof 2 (zi)wf 2 (z 2 ) 

• trH^ 2 1 (z2)MH^ 2 1 (zi)E trFr 1 (z 1 )Fr 1 ( Z2 ) + O p (l), 

which complete the proof. □ 

Remark 4.9. Let Hi = Hi (z) = (1 — z)zafl — za n Tjy. We conclude from the above 
arguments and the fact \wf 2 — mf\ = 0(n _1 ) that 

(4.42) et¥T\ z )e k = e* k U^(z)e k + O p (n^ 2 ), 

and 

itrF- 1 (2i)E,(r- 1 fe)) = EoitrFr'^OEifF- 1 ^)) + 0,(1) 

_ (i-^(i-^Kc) , , 

n 

+ {l ~ = Zl){l -J 2)wfiZl)mf{Z2 h r^\z^ + ap(l), 

which implies 
(4.43) 

i/ere we /iawe wsed £/ie /aci #ie denominator of (|4.43|) is bounded when min{#i,# 2 } > 0. 

4.2. Proof of Lemma 13. 1L Now recalling the contour of integration C, we know it con- 
tains four segments: two horizontal lines and two vertical lines. We need to calculate the 
limit of S n \(z) at the four segments respectively. First of all, considering the top horizontal 
line C* = {z £ C : ?R.z £ [ci — 9,c r + 6], Qz = 6}, we know that there exists some event Q n 
with P(Q„) -> 1 such that, 

E|s„(*) - s n (z)/(Q n )| < (9z)~ 1 P(Q^) -> 0. 

In this part we let Q = Q n = {||(S n + anTN^W < C} with some C < oo. By ()1 .4[) we 
have that for any I > 0, P(Q C ) < It is known that \f+ a " T " > xf' +anTN > \^+ a " T ^ 
for any which implies 



>,)■ 



Here Q* = {||(Sj + a n Tjv) _1 || < C} and = {||(Sy + c^TV)" 1 !! < C}. Notice that we 
also have 

P(Q£) < n~ z and P(Q?,) < n"'. 
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Now we rewrite S n \{z) as S n \ = + + o p (l) with 

s nl =p{ s n{z)I{Q) -M [s n (z)I(Q)]j covariance part 

S$ =p(m Q s n (z)I(Q) - s (2)J(Q)) mean part 

4.2.1. The covariance part. The martingale decomposition used in the proof of Lemma 
14.11 gives that 

n 

S$ = X>i - Ei-OtrCD" 1 - Dr 1 )/(Q i ) + 0p (l) 
i=i 

n n 

= ^(E i -E i _ 1 )tr(S-S l )F^ 1 (z)/(Q l )+^(E i -IE i _i)tr(S l +a n T^) (F^ 1 - Fr 1 ) /(Q^+c^l) 

i=l i=l 
n 

= J^(Ei - E i „ 1 )r ?i (z)/(Q l ) -V 1 - V 2 + o p (l), 
i=l 

where 



2?i = ^(E 4 - ^0(1 - z)w i r*F- 1 (z)r l r*Fr 1 ( z )r l I(Q l ) 



i=l 



P 2 = ^(E, - Ei_i)(l - z)tj7 i r i *Fr 1 (z)(S < + a n T N )¥T l (z)Y l I{Q, i ). 

i=l 

Here we used (|4.12p and the fact that 

(4.44) P(/(Gi) / /(G)) < 

Check that 

r'Fr^jr.r'F-^jr, = V f(z) + ^(sJtrFr 1 ^) + (-UrFr 1 ^)) 2 
Applying (|4.4j) . (|4.10p and Lemma 15.61 we obtain 

E|2>x - Y> - E,. 1 )( 2(1 " z)c7 ^ trFri - (1 = Z lMll!i (trF T ^)I( Qi )f = o( l). 
i=i 

Similarly we have 

E|P 2 - V(Ei - Ei_ x )((l - sjtuf ^(z) - 1 ~ Zj [Wl ) ^ trFr^S, + a n T A r))/(Q i )| 2 = o(l), 
L — ' n 

i=l 

where i^(z) = r*Fr 1 ( z )(S i + Q„T 7V )Fr 1 (z)r i - n" 1 trFr 2 (z)(S J + a n T N ). Thus we have 
p(s n (z)-Es n (z))I(Q) 

n 



^E i («) 2 r ?J -(l-z) ro f^(z) 
i=l 

+ (1 " 2) y r)2??t trFri(.)( St + o n )F- 1 (^)) /(ft) + 0p (l) 
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Check that 



n 



which implies 



t^t / f(z)p(s n (z) - Es n (z))I(Qi)dz 
2m J c t 

n 

-J2 / /(*/(Gi)d(l- z)wt(zMz) + Op(l). 



2iri 

Apparently, (EjJ(Qj)d(l — z)wl r (z)rji(z) / dz} is a martingale difference sequence so we 
can resort to the CLT for martingale (see Theorem 35.12 in [8]). By Lemma 15, 41 and (|4.9j) 
we can get 

ft) 4 

E\Ki(z)I(Qi)\ 4 < — — , 
n 

which together with f)4. lOj) and f)4.8|) implies 

E| J(Qi)d(l - z)tof (z)r?. t (z)/^| 4 = 0{5 n ) -> 0. 

fc=i 

This ensures the Lyapunov condition. Thus, it is sufficient to investigate the limit of the 
following covariance function 

(445) - ±, J c} to)/(«)^<M*.«0*i««*. 

where 

n 

Gn(zi,Z 2 ) =^E i _l[E i ((l-ZlVf(^l) 7 7i( 2 l)J(Q i ))E i ((l-^K* r (Z2Mz2)I(Qi)) 
i=l 

From the arguments in [3] we need to show Q n { z i-, z 2) converges in probability. Applying 
(Oil . glOD . ATOP and the fact wf = wf - wfwf^, we have 



G n (z 1 ,z 2 ) = (l-2i)(l-z 2 )^rof(zi) ro f( 2 r 2 )E i _ 1 [Ei(7 ?i (2;i)/(Qi))Ei(7 ?i (2; 2 )/(Q i ))] +o p (l). 
By Lemma 15.51 we have 



E 



i-l 



(4.46) 
(4.47) 



E|xn| 4 - \Ex 2 u \ - 2 



n 

E,_!^ [Ei (F- 1 (z 1 )/(Q,)) JJ E l {F- 1 (z 2 )I(Q i )) .. 



0=1 



Ex?, + 1 



-11 

)1 2 



Ei-itr 



Ei (F" 1 ^)/^))^ (Fr 1 ( 2;2 )I(Q i )) 
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Using (|4.42p we have 

W = ^^^E [(nr 1 ^)),, (nr 1 ^)),,] +o P (i) 

3=1 

It is worthy to remind the reader that in order to satisfy the condition in the last subsection 
we used here the fact 

P(/(Qi) + I(Qij)) < n~ l . 

And by (|4.43|) we have 

^ - 1 - (^i)(i^ 1 )(i-^K( Z1 M(^) trHr i( Z2 )H r 1 (. 1 ) 

From the arguments of the next part we can conclude that for z £ C l 

E s n (z) = s (z) + O(n~ 1 ) ^s(z). 

Thus we get in probability 



Gn(z 1 ,Z 2 ) ->■ (t+1) j - 



f y(l-z 1 )(l-z 2 )Tu(zi)zu(z 2 ) IpY (f\ 
J {{l-z 1 )m-z 1 at)((l-z 2 )m-z 2 at) ul mp\ h l 



+ f y{l-z 1 )(l-z 2 )zu(z 1 )m(z 2 ) ,j?Y (+\ 
1 J ({l-z 1 )m-z 1 at)({l-z 2 )vv-z 2 at) ur m P \ t ') 

- * - / l:t-tt iF ^ I (1 ( -;t2- ( :> ^>> 

which is ([SSD . 

In addition, by definition of S^* we get 

E|S#(*i) - ^^s)! 2 = |*i - ^pEltrD-^^OD" 1 ^) - E trD- 1 (^)D- 1 (^)| 2 /(Q). 
Therefore using (I4,12h , Lemma 14.11 Lemma 14.51 and the fact 

D" 1 ^) = (1 - zy l (l + a n T N ¥- l {z)). 

we can easily check that 

(4.48) E\S$( Zl )-S$(z 2 )\ 2 <C\ Zl -z 2 \ 2 , Zl ,z 2 eC\ 

which implies the sequence {<S^(-)} forms a tight sequence on C*. 

4.2.2. The mean part. From the definition of the Stieltje transform of s n (z) we have 

1 1 

s n (z) = s FBn = -trD" 1 = -tr{S n + a n T N )Y^ l {z) 
P P 

= (1 + — J-trS^F- 1 ^) " - = — trSnF- 1 ^) - -. 

z p z zp z 

Using (|4.1ip we get that 

n 

(4.49) SnF- 1 ^) = J2^i r * F T\ 

i=l 
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which implies 



n 

Wi 



-trS n F- 1 (.) = - f ^( 1 - 1 E 
p p(l — z) \ n *- 

Thus we have 

1 n 

(4.50) - VtJ7i = l-i/ n (l-js)Css n + l), 

i=i 

and 

(4.51) EqvjiUQi) = 1 - y n (l - «)(«Eo*„/(Gi) + 1). 

Denote A n = E (roiJ(Si)) (E (roi/(Qi))I + OnT^) -1 , C n = A n - zl and A(z) 
F.QS n (z)I(Qx) — p~ 1 trC n . Then we obtain that 

(4.52) p -hrC n = f WCQQ+ant T 



Recalling the definition of ruo and (|2.9p we have 

l-^o 1 1 + J_ /" ^ ^ (t)j 



zy(l — z) z 1 — z 1 — z J (I — z)wq — za n t 

which implies 

(1-z) 



™ =(l + y ~7-\ — i -dF^{t) 

(1 — z)wq — za n t 



-i 



According to (|4.5ip and (|4.52p we get that 

E w 1 I(Q 1 )=(l + y [ (1 ~ ^ -dF T ^(t) + (E w 1 I(Q 1 )r 1 zy(l-z)A n ) 

V J (1 - z)E roii(Si) - z«nt / 



The difference of the above two identities yields 

\2 



w T(n , f E Q w l I(Q l )w {) y(l-z) 2 (w () -E w 1 I{Q 1 )) T 

tz7 - hwil(Qi) = / — — — — — — -dF N (t) 

J [{1 - z)w - za n t\[(l - z)E Q vJiI{Qi) - za n t\ 

+ zu zy(l - z)A n . 
Thus we use (|4.44|) to obtain that 
(4.53) 

E S n (z)I(Q) - S {Z) = W A n 1 - / T7Z r -777- 777 — — -rdF 

V J [(1 - z)m - za n t\ [(1 - z)E miI{Qi) - za n t\ 
We will use the following lemma. 

Lemma 4.10. For z G C t 

a / x (rrix - t - 2)a n (l - z)(wf) 2 TT i . . o i , m 

pA z = ^ ^ ^JJ_trHo 1 (z )trU 2 (z )T N 

pn 

+ t( i-,)K)^. 
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Proof. It follows from the definition of D n and C n that 

D n X - C n X = C n ( A n ~ ^n)F>~ 1 = C^AnD; 1 - C~ 1 B n D~ 1 . 

Using ()4.49p we have 

n 
i=l 

and 

n 

C- 1 A(B n -zI)- 1 = C- 1 Aj2mr i r*Fr\z) + a n C- 1 AT N F- 1 (z). 

i=l 

Then from the definition of A(z) and (|4.1ip we have 

P A n = nE ro 1 r* 1 F^ 1 (z)C~ 1 Ar 1 /(Q 1 ) 
+E a n trAT N F~ 1 (z)C~ 1 I(Q 1 ) - nE oroi r*F- 1 (z)C^ 1 r l /(Q 1 ) 
= di + d 2 + d 3 + d 4 , 

where 

di =nE roi/(Qi)r^F^ 1 (2)C" 1 Ari -E CT 1 /(Qi)E trF^ 1 (z)C- 1 A, 
d 2 =E roiJ(Qi)E trFJ" 1 (^)C -1 A - E w 1 I(Q 1 )E o tTF- 1 (z)C- 1 A, 
d 3 =E oroi /(g 1 )E trF^ 1 (z)CT 1 - nE w 1 I(Q 1 )r* 1 F^ 1 (z)C- 1 r 1 , 
d 4 =E roi/(Q 1 )E trF" 1 (z)C- 1 - E roi/(Si)E trFr 1 {z)C~ 1 . 
Firstly consider d\. We apply f|4.3j) and (|4.2p to represent di as 

(4.54) di = -n(l - ) 2 E r?i (r^F^C^Ari - n -1 trFi 1 C -1 A) I(Qi) 

(4.55) +(1 - z)(wf ) 2 E £i (trF^C^A - EotrF^C^A) !(&) 

(4.56) +n(l - z) 2 (rof ) 2 (EowiTfrjF^C -1 An - ra^EowiT^lxF^C^A) /(&). 

Note that similar to (14.181) we can get that ||C _1 || and ||AC _1 || are both bounded when 
z G C t . Thus by Lemma 14.11 and Lemma 15.41 we obtain that 

E|trF^ 1 (z)C- 1 A - E trF^ 1 (z)C~ 1 A| 2 = 0(1), 

E|r^F^ 1 (z)C" 1 Ari - n~ 1 trFj" 1 (z)C~ 1 A| 2 = 0(n _1 ). 

These together with (|4.10p . (|4.8I) . (I4.14p and Holder's inequality imply that 

<|OH|> = O p (ra~ 1/2 ) and (14561) = O p (<5 2 ) 

Using Lemma (15. 5p we have 

g3t = -K - t- 2)(1 - 2 )( ro f) 2 i,„Eo(Fr 1 (2)) ii (F7 1 ( 2 )C- 1 A) n /(Q 1 ) 

_ (t+ i)(i-,)K)' 2 " 
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For d,2, we use (14. 12H to get 

d 2 =(1 - z)EoW!E ( roi /(Q 1 )r*Fr 1 (^C- 1 AFr 1 (^)/(Qi)r 1 ) 

= (i-^K) 2 trF „ 2( ^ )c _ lA/(Qi) + Qp{i) 

Similarly we can get 

da = (m» - t - 2)(1 - z)(wf ) 2 y„E (f^M) n (f^'MC -1 ) ^/(Qi) 

+ (t+ i)(i-,) ( ^ 2 11 

n 

and 

d4 = _ (lz^il! trFr 2 (z)c -i + 0p(1) . 

Therefore combining the above four equations we conclude that 
E trD; 1 /(Q 1 )-trC; 1 

= ( ^ " m n ~ ^ ( ^ Eo(F r ^(,)) ii (F r ^(,)C^(I - A)) i/ (Q 1 ) 

+ t(1 ~ ^^^ EotrF^^C-^I - A)/(Q 1 ) + o p (l) 
n 

By Lemma 14,31 -Lemma 14.51 and the fact that 

IIC-^I-A))! = a n Tjv((l- z)E w 1 I(Q 1 )I -z^Tjy)" 1 <C 

we have that 

E trD- - trC? = = t = ^ - trU^(z)trU^ { z)T N 



pn 



+- ^ 11 - trH- 3 (z)Tjv + o p (l) 

which complete the proof of this lemma. □ 

Noting the transform s n (z) = j^s n (j^) - s (z) = (jf^o^) " 

and (3.12) in [2] we have that for z G Ct 

1 _ f a n y n z{l-z)t dpTN \ 1 ^ ^ 

J [(1 - z)Ezui - za n t][(l - z)zu - za n t] J 

Thus we have Ks n = sq + 0(n _1 ) — > s, which combined with (|4.53p gives (|3.7p . 

We so far have proved Lemma 13.11 under the condition that z G C*. It is easy to 
check that the above arguments evidently work when z belongs to the bottom line due to 
symmetry. 

When z belongs to the left vertical line of the contour, that is z G C l = {Kz = q— 9, G 
[—9, 9]}, we split C l into two parts C[ + C 2 where 

C[ = = Q - 9, n- l e n < |9fz| < 0} 
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and 

C\ = {Rz = ci-9, \Sz\ < n _1 e„} 
with e n = n~P for some (3 G (0, 1),. We truncate s n at each part, that is 



SnO) 



s n (z), z G C{; 

s n (Kz + in~ 1 e n ), z E C l 2 . 



Then from a similar argument in [3] we can get that the limit of p(s n (z)I(Q) — sq) has the 
same form as Lemma I3TT1 provided. Here Q = {||(S n + a n TAr) _1 || < C} Hl^i™ > c i ~ L } 
with small enough i > 0. And the situation is the same if z belongs to the right vertical 
line of the contour due to symmetry. We omit the details. 

5. Some basic lemmas 
In this section we give some basic lemmas which are used in the paper. 
Lemma 5.1. (Lemma 6.1 in [18j ) 

«(*)(■« (*) + !-*/) 



s(z) 

(Hz))' - 



(■*(*) + 1/(1 -y))(i-yy 

(g(z) + i/(i-y))(i-y) 
*(*)(■*(*) + i) 

( ^ (z) + l / (l-y) )2(1 _ y) 2 
(l-y)s(z) 2 + 2s (*) + !-*/' 



i y ( 2 ) 

+ t) dF mp (t) = ( 3 (z) + 1/(1 _y ))(1 _ y) ' 



mpl ; (l-y)a(z) 2 +2a(«) + l : 



= - (1 r^'^fv^'Ii^^ ^^y = -(i - y(« (*)) 2 c» (*) + i) 2 )s- 2 (z)(s(z)y 

(s(z)Y{s(z) + l) 2 

2y/«t(^)) 3 (^)+t)- 3 ^ p (t) , ((l-YYs{z) 2 + Ts(z) + l-y 

— = log 



(l - y / (5(z)) a + 1)" 2 



(l-y)'s(^) 2 + 2's(z) + 1 



((i-y(s(z)) 2 (s(z) + i) 2 )) 

Lemma 5.2. (Lemma 2.3 in |14j ) Letx,y be arbitrary non-negative numbers. For A and 
B square matrices of the same size, 

F V( ABKAB )*{( xy , oo)} < F^^x, oo)} + oo)} 
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Lemma 5.3. (Lemma A. 45 and Corollary A.41 in Let A and~B be two nxn Hermitian 
matrices. Then 

L (F A ,F B ) < ||A - B||, 

and 

L 3 (F A ,F B ) < -tr(A - B)(A - B)*, 
where L(-,-) denotes the Levy distance and \\ ■ \\ denotes the spectral norm. 

Lemma 5.4. (Lemma 9.1 of [4],) Let A be an n x n nonrandom matrix bounded in norm 
by M, and X = (xi, . . . , x n )* be a random vector of independent entries. Assume that 
Kx-i = 0, E|xj| 2 = 1, E|xj| 4 < 00 and \x%\ < b n \fn with 5 n — > slowly. Then for any given 
2 < I < 61og(n<5 2 ) with some b > 1, there exists a constant C such that 

E\X*AX - trA\ l < n^nS^iMCSl) 1 . 

Lemma 5.5. ((1.15) of Let A = (dij) pXp and B = (bij) pXp be nonrandom matrices 
and X = (x±, . . . ,x n )* be a random vector of independent entries. Assume that Ex, = 
and K\xi\ 2 = 1. Then we have, 

E(X*AX - tr A)(X*BX - trB) 

p 



(5.1) 



I Ex 



2|2 



2)0*4^ + trA x W x + tr AB, 



i=l 



where A x 
matrix. . 



Ex 



i ai i)pxp> 



B, 



Ex?6jj) x and the superscript T is the transpose of a 



Lemma 5.6. (Burkholder Inequality) Let {X^} be a complex martingale difference se- 
quence with respect to the increasing a-field Fk, and let E/% denote conditional expectation 
with respect to T^. Then we have 
(a) forp > 1, 



(5.2) 

(b)forp>2, 



E 



k=l 



p/2 



\k=l 



(5.3) 
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